System for visualization and control of surgical devices utilizing a graphical user interface

ABSTRACT

A system for visualizing and guiding a surgical device having a first imaging device of a first type and has a first image output. The first imaging device is positioned to image an area being subject to surgery. A second imaging device of a second type has a second image output. The second imaging device is positioned to image an area being subject to surgery. A computer is coupled to receive the first and second image outputs and a computer software program, resident in the computer receives and displays information received from the surgical device and/or for guiding the operation of the surgical device and for generating a graphic user interface including selectable menu and submenu items. The surgical device is coupled to the computer.

FIELD OF THE INVENTION

The invention relates to a system for the control and visualization ofmedical devices positioned in a patient's body for ablation of a tumor,such as a uterine fibroid and, more particularly, to a user interfacefor visualizing an ultrasound image and dynamic 3D avatar guidance of anablation probe during a surgical procedure, enabling the operator tomake operational decisions based on those visualizations.

CROSS REFERENCE Background of the Invention

Advances in technology have resulted in graphical user interfaces thatallow a practitioner or other medical professional to visualize aplurality of images obtained from multiple medical devices such as alaparoscopic cameras and ultrasound probes together on one screen.Current systems use simple picture in picture technology to view smallerlaparoscopic camera images and ultrasound images on a larger guidancesystem screen.

SUMMARY OF THE INVENTION Brief Description of the Drawings

FIG. 1 illustrates an overview of the operating room layout useful withthe inventive system;

FIG. 2 illustrates an ablation system incorporating computer controls inaccordance with the inventive system;

FIG. 3 shows the inventive graphical user interface screen in which thedisplay is divided into two separate screens with an associated text boxand a control tab;

FIG. 4 illustrates the inventive user interface being displayed from thesurgeon's point of view with the ultrasound beam facing the patient'shead;

FIG. 5 illustrates the inventive user interface being displayed from thesurgeon's point of view with the ultrasound beam facing the patient'sfeet;

FIG. 6 illustrates the inventive user interface being displayed from thelaparoscopic point of view with the ultrasound beam facing the patient'sright hand;

FIG. 7 illustrates the inventive user interface being displayed from thelaparoscopic point of view with the ultrasound beam facing the patient'sleft hand;

FIG. 8 shows an overhead view of a patient's body divided into fourquadrants and the icon depictions associated with each quadrant;

FIG. 9 shows the graphical user interface screen in which thenavigational tool has scrolled to the tool icon control button;

FIG. 10 shows the graphical user interface in which the tool iconcontrol button has been selected;

FIG. 11 shows the graphical user interface screen in which thenavigational tool has scrolled to the three dimension icon controlbutton;

FIG. 12 shows the graphical user interface screen in which thenavigational screen is displayed in three dimension;

FIG. 13 shows the graphical user interface in which the navigationaltool has scrolled to the control button;

FIG. 14 shows the graphical user interface in which the navigationaltool has scrolled to the grid icon control button;

FIG. 15 shows the graphical user interface in which a grid is displayedon the GUI;

FIG. 16 shows the graphical user interface in which the navigationaltool has scrolled to ultrasound probe icon control button;

FIG. 17 shows the graphical user interface in which the ultrasound datahas been made full screen and the navigational screen has beeneliminated.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 illustrates an overview of the typical operating room layout forthe present invention. The surgeon 10 is standing on the left hand sideof patient 12 who is lying on operating table 14. Surgeon 10 may alsostand on the right hand of patient 12, whichever is most convenient. Twoscreens 16, 18 are located on the opposite side of patient 12 fromsurgeon 10. Screen 16 displays video from a laparoscopic camera andscreen 18 displays the inventive user interface. Screens 16 and 18 canbe any type of display monitor such as a computer monitor, television,handheld device screen, etc.

FIG. 2 illustrates an exemplary system for implementing the presentinvention. Computer 20 may be any control device, such as amicroprocessor, personal computer, or a more powerful or less powerfulcomputer with typical computer-type operating system. Computer 20includes display screens 16 and 18, which may optionally be atouchscreen to provide a second means of navigation.

Personal computer 20 also incorporates software 22. Software 22 may beof any type for use on any suitable computing device, and which may beeasily written by a programmer of ordinary skill in the art who isinformed by this specification. The software is responsive to produceimages illustrated in the drawings and stored in memory 24 of computer20. The software performs navigation functions by being responsive totouchscreen entry on display screen 18. Likewise, in response tooperation by touching display screen 18, computer 20 may cause thescreen to change in one of the ways described in full detail below.

Computer 20 communicates with a plurality of medical devices 26 throughan interface board 28. Medical devices 26 include an ablationinstrument, laparoscopic camera, and ultrasound probe, or any otherinstrument useful in imaging and treating uterine fibroids or otherpelvic tumors. At the same time, medical devices 26 provide informationto interface 28 which in turn provides this information to personalcomputer 20 for display on display screens 16 and 18.

FIG. 3 illustrates the preferred embodiment of the inventive userinterface. Software takes data from transducers located on an ablationprobe, and data from transducers on an ultrasound probe and displays itto a user via user interface 100 on display screens 18. Screen 18displays the inventive user interface 100. Video from a laparoscopiccamera is communicated directly to display screen 16. User interface hasan overlay 102 that provides the appearance of a physical bezel thatdivides the display into two screens 104, 106. Overlay 102 has a controltab 108 located on the left hand side of overlay 102. Control tab 108has five selections to choose from 110, 112, 114, 116, 118.

Overlay 102 includes two text boxes 120, 122. Text box 120 is locatedunder screen 104 and displays information relating to what is displayedon screen 104. For example, text box 120 may display the depth of theultrasound.

Text box 122 is located under screen 106 and displays informationrelating to what is displayed on screen 106. For example, text box 122may display the point of view of the image displayed on screen 106.

Overlay 102 has a meter 124 located in between screens 104, 106. Meter124 indicates the distance between the tip of an ablation probe to theplane of an ultrasound scan. Meter 124 has hash marks 126 to indicatethe distance from the tip of the ablation probe to the plane of theultrasound scan. The central hash mark 128 is green and indicates thatthe tip of the ablation probe is in line with the plane of theultrasound scan. The hash marks above central hash mark 128 are blue andindicate that the ablation needle is behind the plane of the ultrasoundscan. The space between the hash marks increases the further away fromcentral hash mark 128 you get. The hash marks below central hash mark128 are yellow and indicate that the ablation needle is in front of theplane of the ultrasound scan. Meter 124 also has a sliding hash mark 130that slides up and down meter 124 indicating the dynamic location of thetip of the ablation needle relative to the plane of the ultrasound scan.Sliding hash mark 130 is translucent, allowing the user to see the blue,yellow, or green hash mark underneath.

Screen 104 is used to display a virtually complete ultrasound screen intwo dimensions. Screen 104 contains a photorealistic avatar ofultrasound shaft 132. Ultrasound shaft 132 is placed above the displayof ultrasound beam 134 in order to provide orientation for ultrasoundbeam 134 to the user. This enables the user to easily see the directionthe ablation probe is entering the ultrasound beam.

Screen 106 also displays icon 136, which indicates the direction theultrasound beam is pointing in relation to the uterus of patient 12.

Icon 136 may be a depiction of a right hand with the letter “R” toindicate that ultrasound beam 134 is pointing towards the right hand ofpatient 12.

Icon 136 may be a depiction of a left hand with the letter “L” toindicate that ultrasound beam 134 is pointing towards the left hand ofpatient 12.

Icon 136 may be a depiction a person's head to indicate that ultrasoundbeam 134 is pointing towards the head of patient 12.

Icon 136 may also be a depiction of feet to indicate that ultrasoundbeam 134 is pointing towards the feet of patient 12.

Icon 136 will change in real time between these depictions depending onthe direction ultrasound beam 134 is pointing. For instance, if a usermoves an ultrasound beam from pointing towards the feet of a patienttowards to the right hand of the patient, icon 136 will change from adepiction of feet to a depiction of a right hand with the letter R. Thisis done automatically without any input from the surgeon. The surgeonalso has the capability to freeze the orientation view if desired.

FIG. 4 illustrates the inventive user interface with the ultrasound beamfacing the patient's head in the surgeon's point of view. Icon 136displays as a depiction of a human face in profile.

FIG. 5 illustrates the inventive user interface with the ultrasound beamfacing the patient's feet in the surgeon's point of view. Icon 136displays as depiction of a human foot.

FIG. 6 illustrates the inventive user interface with the ultrasound beamfacing the patient's right hand in the laparoscopic point of view. Icon136 displays as a depiction of a right human hand with the letter “R.”The letter “R” can be displayed inside the depiction of the human handor alternatively, in the immediate surrounding area of Icon 136, such asto the right, to the left, on top of, or beneath, Icon 136.

FIG. 7 illustrates the inventive user interface with the ultrasound beamfacing the patient's left hand in the laparoscopic point of view. Icon136 displays as a depiction of a left human hand with the letter “L.”The letter “L” can be displayed inside the depiction of the human handor alternatively, in the immediate surrounding area of Icon 136, such asto the right, to the left, on top of, or beneath, Icon 136.

The display of ultrasound beam 134 will also rotate direction dependingupon the orientation of ultrasound beam 134. For example, in FIG. 4,ultrasound beam 134 is pointed at the patient's head, thus icon 136 willbe a depiction of a head ultrasound beam 134 will point to the right.When the ultrasound transducer is physically rotated so that it pointsto the patient's right hand, the transducer shaft 132 will occlude theimage of ultrasound beam 134. In order to provide a user with a view ofthe system, the system will virtually rotate ultrasound beam 134 back tothe right and icon 136 will display a right hand, to signify thatultrasound beam 134 is pointing to the patient's right hand, as seen inFIG. 6. This happens whenever ultrasound shaft 132 is physically rotated90 degrees as shown in FIG. 8.

FIG. 8 illustrates which graphical depiction Icon 136 will display whenthe ultrasound probe is in four defined quadrants of patient 12. Thedepiction is based on the location of the ultrasound probe in patient 12in relation to the uterus of patient 12. Patient 12 is divided by acenterline 310 y-axis with the patient's head being 0 degrees and thepatient's feet being 180 degrees. Patient is then divided by an x-axisthrough the uterus. Patient 12 is then further divided into four 90degree quadrants 302, 304, 306, and 308.

Quadrant 302 is defined between 315 degrees to 45 degrees measured fromthe uterus. When the ultrasound probe is in quadrant 302, Icon 136 willdisplay a depiction of a human foot since to view the uterus theultrasound beam would have to be directed towards the patient's feet.

Quadrant 304 is defined between 45 degrees and 135 degrees. When theultrasound probe is in quadrant 304, Icon 136 will display a depictionof a right human hand with the letter “R”.

Quadrant 306 is defined between 135 degrees and 225 degrees. When theultrasound probe is in quadrant 306, Icon 136 will display a depictionof a human face in profile.

Quadrant 308 is defined between 225 degrees and 315 degrees. When theultrasound probe is in quadrant 308, Icon 136 will display a depictionof a left human hand with the letter “L”.

Referring back to FIG. 3, screen 106 is used to display ablation probeguidance information in either two dimensions or three dimensions.

Again referring to FIG. 3, overlay 102 has a control tab 108 containingcontrols 110, 112, 114, 116, and 118 located on the left side of overlay102 which can be used by a surgeon or medical professional whilemaintaining the sterile field.

Control 110 has a depiction of a tool, and when selected allows amedical professional to change the display of the system.

Control 112 displays “3D,” and when selected allows a medicalprofessional to view the guidance information on screen 106 in threedimension.

Control 114 has a depiction of a lock, and when selected allows amedical professional to freeze the point of view being displayed.

Control 116 has a depiction of a grid, and when selected displays a gridover the ultrasound data, which decreases the need for measurement ofthe dimensions of any tumors or masses being imaged.

Control 118 has a depiction of an ultrasound probe, and when selectedbrings screen 104 into full-screen, eliminating screen 106.

Selecting control 110 (FIG. 9) with navigational tool 111 causes thesystem to exit the display of FIG. 2 and go to the display of FIG. 10.

Selecting control 112 (FIG. 11) with navigational tool 111 causes thesystem to exit the display of FIG. 2 and go to the display of FIG. 12.The display of FIG. 12 is substantially the same as the display of FIG.2, except that screen 106 is now viewed as three-dimensional as opposedto two-dimensional.

Selecting control 114 (FIG. 13) with navigational tool 111 causes thedisplay in FIG. 2 to become locked in the currently displayed point ofview. Thus, the display in screen 104 will not automatically switchpoint of views based on the placement of the ultrasound probe. Instead,the screen will stay locked in the point of view at the time ofselection of control 114.

Selecting control 116 (FIG. 14) with navigational tool 111 causes grid138 to be displayed over screen 104 as illustrated in FIG. 15. Grid 138decreases the need for measurement of the dimensions of any tumors ormasses being imaged.

Selecting control 118 (FIG. 16) with navigational tool 111 causes thesystem to exit the display of FIG. 3 and go to the display of FIG. 17.FIG. 17 illustrates a full screen view of the ultrasound data of screen104, eliminating everything else from the display except for overlay142, which contains control tab 108 with controls 110, 112, 114, 116,and 118. Overlay 142 is substantially similar to overlay 102 inappearance except without the division into two screens.

It will be appreciated by those skilled in the art that changes can bemade to the embodiments described above without departing from the broadinventive concept thereof. It is understood, therefore, that thisinvention is not limited to the particular embodiments disclosed, but itis intended to cover modifications that are within the spirit and scopeof the invention, as defined by the appended claims.

What is claimed is:
 1. A system for visualizing and guiding a surgicaldevice, comprising: a laparoscopic camera device having a first imageoutput, said laparoscopic camera being positioned to image an area beingsubject to surgery; an ultrasound device having a second image output,said ultrasound device being positioned to image said area being subjectto surgery; a computer coupled to receive said first and second imageoutputs; and a computer software program, resident in said computer forreceiving and displaying information received from said surgical deviceand for guiding an operation of said surgical device and for generatinga graphic user interface including selectable menu and submenu items anda display having a first display area and a second display area, whereinsaid display shows an avatar of said ultrasound device positioned abovean ultrasound image rotating within said second image output such thatthe ultrasound image becomes occluded by the avatar of said ultrasounddevice when said ultrasound device is physically rotated relative tosaid area being subject to surgery, the ultrasound image rotating in amanner corresponding to a rotation of the ultrasound device, and whereina relative positioning of the avatar of said ultrasound device relativeto an avatar of an ablation probe provides an indication of a directionof the ablation probe entering the ultrasound image and wherein therelative positioning further provides a depiction of a relative quadrantof the patient in which the ultrasound device is positioned.
 2. Thesystem of claim 1 further comprising: a menu with selectable graphicalobjects; a photorealistic graphic of said surgical device; an iconindicating a direction of said photorealistic graphic of said surgicaldevice in relation to a patient's body; at least one region for thedisplay of data from said surgical device; and a meter located inproximity of said first and second display areas, which indicates adistance between said surgical device to the plane of said ultrasounddevice.
 3. The system of claim 2, wherein said icon changes dependent onthe direction said surgical device is positioned in relation to saidarea being subjected to surgery.
 4. The system of claim 3, wherein saidicon changes occur at fixed points measured in degrees with 0 degreesbeing indicative of a direction towards the centerline of the head ofthe patient being subject to surgery, 90 degrees being indicative of thedirection towards the patient's left hand, 180 degrees being indicativeof the direction towards the patient's feet, and 270 degrees beingindicative of the direction towards the patient's right hand.
 5. Thesystem of claim 2, wherein said meter is comprised of an emboldenedcenter hash mark, hash marks of a first color above said emboldenedcenter hash mark indicating the surgical device is behind a plane of theultrasound device, and hash marks of a second color below saidemboldened center hash mark indicating the surgical instrument is infront of the plane of the ultrasound device.
 6. The system of claim 1,wherein said first display area displays input from said ultrasounddevice, and a photorealistic avatar of an ultrasound probe.
 7. Thesystem of claim 2, wherein said computer when executing said computersoftware program automatically rotates the point of view of saidphotorealistic graphic of said surgical device when said surgical deviceis positioned so as to block said ultrasound image within said secondimage output.
 8. The system of claim 2, wherein said meter furthercomprises a sliding hash mark that is coupled to a sensor on saidsurgical device, said sliding hash mark being capable of sliding up anddown said meter in relation to a location of said surgical device tosaid ultrasound device.
 9. The system of claim 1, wherein said seconddisplay area displays image guidance information for said surgicaldevice.
 10. The system of claim 1, wherein said display shows atwo-dimensional or a three-dimensional image.
 11. The system of claim 1,further comprising said surgical device coupled to said computer.